The present invention generally relates to a procedure for determining the quality of a quantity of properties describing a machine. The machine of the type under consideration can be both software and hardware. Within the meaning of this application a machine is suitable to be described by means of properties. Such properties are generally known to the person skilled in the art of functional verification of digital circuits. An exemplary machine is thus a digital circuit.
When performing functional verification, an RTL-description of a circuit is verified as to whether it will function properly. The quality of the RTL-description after functional verification is decisive for the success of a circuit design project. Remaining errors are likely to cause high costs, loss of time and reputation.
In addition, using simulation as the principal work horse in functional verification is standard. For this, input patterns are preset, either automatically or by the user, and the output patterns generated by simulation compared with expectations. Errors are detected if an output pattern does not meet with expectations. Automatic mechanisms for error detection in simulation are hereinafter called monitors.
In order to detect an error, there must be given a suitable input pattern for which the simulation produces an output pattern in which the error shows up. Even in small circuits the quantity of all possible input patterns is very large. In addition, the factor between the run time in the real circuit and its simulation is in the range of 106. Therefore only a few input patterns can be simulated in comparison with the total number, so that many functional errors remain undetected.
Increasingly so-called property checkers are used for functional verification. Property checkers are given, as something input something called a “property”, i.e. a relationship concerning certain aspects of the circuit, that is described in a formal language. Within surprisingly short time, the property checker yields a result in the form of
either the information that the property has been proven by the property checker. In this case the relationship between the behavior aspects which is described in the property will always occur, irrespective of how the circuit is operated
a rebuttal in which an input pattern for the circuit is described that violates the relationship between the behaviour aspects. This rebuttal is generated by the property checker.
The automatic generation of counter-examples and the rapid checking of the property for all possible input patterns constitute a considerable advancement in the functional verification vis-à-vis simulation.
For this reason, property checkers are also being implemented today by innovative verification groups. However, they only verify selectively as behavioural associations that are considered critical, which were chosen by a verification engineer based on his/her intuition and experience. This nonetheless only enables gradual improvements in the quality of the RTL description after verification. A relatively high number of errors in the design are still present in the result.
In the area of circuit verification by means of simulation, suggestions have already been made concerning procedures for the evaluation of the quality of the verification environment, which even calculate measures, which are supposed to be related to this quality. In this sense, the number and coverage of the input patterns used for simulation is, on the one hand, of the greatest importance for the quality of the verification environment and, on the other hand, the effort to achieve a multiplicity of different types of input patterns is also cause for a large resource requirement.
The corresponding procedures have already been in use for years and are supported by commercial EDG tools (see e.g. D. Dempster, M. Stuart: “Verification Methodology Manual—Techniques for Verifying HDL Designs”, Teamwork International, 2nd edition, 2001; and S. Tarisan, K. Keutzer: “Coverage Metrics for Functional Validation of Hardware Designs”, IEEE Design & Test of Computers. 2001.) The metrics for determining the measures are often code-based such as the line coverage, path coverage, assignment coverage and coverage of states for explicit state machines from the RTL code.
Such methods for measuring the quality of a verification were first developed for software verification. Unlike sequential software, however, hardware operates in the form of parallel processes. The dependencies generated by this are therefore not taken into account by the methods, so that the resulting quality measures are not reliable. Therefore, alternative measures have been suggested, for example those based on signals and require, for example, the assignment of all possible values. In addition, there are functional metrics, which display how often a certain functionality has been executed.
An inherent weakness in all of these approaches exists, however, in that attaining a level of 100% is far from securing the entire verification of all functionality. On the one hand, these approaches already achieve 100% coverage even when not all functionality and consequently all possible error sources in the circuit tested have been injected; on the other hand, there is no systematic test which shows whether the verification environment is capable of actually identifying every error that shows up.
The property check is described in exemplary terms in: A. Biere, A. Cimatti, M. Fujita, Y. Zhu: “Symbolic Model Checking using SAT procedures instead of BDDs” Proc. of 36th Design Automation Conference, 1999; and in: “Formale Verifikation für Nicht-Formalisten (Formal Verification for Non-Formalists)”, Bormann, J. and Spalinger, C: (2001), Informationstechnik und Technische Informatik, Vol. 43, Issue January 2001, Oldenburg Verlag.
When performing the quality assurance of a verification environment based on the property check, the issue is not necessarily the multiplicity and diversity of the input patterns, since a property checker functions as if it were inspecting all input patterns. It is rather the integrity of the property set, that is, whether each error in the circuit is detected when proving at least one of the properties.
Papers in this field are documented, for example, in: Hojati, R.: “Determining Verification Coverage Using Circuit Properties” U.S. Pat. No. 6,594,804, granted Jul. 15, 2003; Hoskote, Kam, Ho, and Zhao: “Coverage Estimation for Symbolic Model Checking” Proc. of 36th Design Automation Conference, 1999; and in: Hoskote, Y. “Property Coverage in Formal Verification”, Patent WO200079421-A2. These approaches are all based on sequentially injecting errors in the circuit and then checking whether at least one of the properties is disproved by the property checker in the circuit that has been modified in this way. If each of the errors injected ends up being disproved, the quality of the property set will be designated as sufficiently high. Otherwise, a measure concerning the proportion of those injected errors will be determined which were detected by the property set.
With the known procedures, it is either injected as an error that signals are inverted in various achievable states or the gate, at whose output the signal to be covered is picked up, is changed on the net list.
These procedures are heuristic. There is no guarantee that errors other than those injected will be detected by the property set. In addition to the quantity of properties, the procedures also require the circuit description.
Task
Since so far the properties, as described above, have only been applied in a selective manner and based on experience, a need exists for a procedure, which can reliably specify the quality of a quantity of properties in a reproducible manner, particularly in the form of a measure which can then be brought into accordance with a target value by expanding or adjusting the volume.
Furthermore, a need exists for procedures for more reliable verification and the specification of circuits based on the reproducible quality of a quantity of properties.
Consequently, one object of the invention in question is to specify procedures which determine and utilize the quality of a quantity of properties describing a machine without having to resort to excessive use of resources or experience values.